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ABSTRACT 

One of the critical components in the curriculum is 
evaluation of students' knowledge or the amount they learned in the course. 
Over time, the assessment process has not changed significantly; it is only 
recently that university instructors are turning to technology for assistance 
in this labor-intensive task. There are disadvantages and advantages for both 
instructors and students of online testing. Much of the discussion 
surrounding alternative testing implementation centers on various learning 
styles. One question is: Why do we continue to assess students in traditions 
set long ago? Some other questions are raised: Does the online testing 
improve student learning and achievement? Can/should instructors tailor tests 
to student's preferences for learning? How does the creation of online exams 
affect the instructor' s workload? This research looks at two traditional 
face-to-face Information Systems courses which incorporated online testing 
into their curriculum and begins to provide some answers to these questions. 
(Contains 18 references.) (Author) 
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ABSTRACT 

One of the critical components in the curriculum is evaluation of the student's knowledge or the amount they learned 
in the course. Over time , the assessment process has not changed significantly; it is only recently that university 
instructors are turning to technology for assistance in this labor-intensive task. There are disadvantages and 
advantages for both instructors and students of on-line testing. Much of the discussion surrounding alternative 
testing implementation centers on various learning styles. One question is “why do we continue to assess students 
in traditions set long ago? ” Some other questions are raised: Does the on-line testing improve student learning and 
achievement? Can/should instructors tailor tests to student's preferences for learning? How does the creation of 
on-line exams affect the instructor's work-load? This research looks at two traditional face-to-face Information 
Systems Courses which incorporated on-line testing into their curriculum and begins to provide some answers to 
these questions. 



INTRODUCTION 

“On-line education can be defined as teaching and 
learning activities enabled by electronic media” (Romm 
and Taylor, 2000). Electronic media or “technology is 
not simply an add-on service as computers or 
audiovisual...” (Tsichritzis 1999, 93). Emerging 
technologies are an integral part of the university 
curriculum — not only are they, themselves, studied, but 
also the manner they support the delivery of many facets 
of the curriculum. One of the critical components in the 
curriculum is evaluation of the student’s knowledge or 
the amount they learned in the course. Over time, the 
assessment process has not changed significantly (Natal, 
1998); it is only recently that university instructors are 
turning to technology for assistance in this labor- 
intensive task. 

There is a growing emphasis on accountability offaculty 
and an increased importance on ensuring student 



success, where success is being defined as student 
learning. AACSB reviewers are evaluating business 
curriculum with a strong interest in outcomes assess- 
ment — measures of student learning. The questions 
being asked are- are the students learning what the 
university thinks they are learning and can it be 
measured? How is improvement in student learning 
measured? Is the testing environment secure? The 
answers to these questions are being confounded by the 
proliferation of distance education or on-line courses 
using the Internet as a means of delivery. 

For a course that is taught fully on-line or distance, it is 
expected that the testing component is also 
accomplished at a distance or on-line. However, on-line 
testing or assessment may be an appropriate venue for 
courses taught in a face-to-face environment. It may 
relieve part of the repetitious, time-consuming activities 
involved with creating exams, grading exams, and 
calculating the exam scores and final grades. More time 
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might be spent in the actual teaching of the course 
content and less on administering exams. This research 
looks at incorporating on-line testing into a traditional 
lecture-based course. 

BACKGROUND 

Most of the on-line testing literature involves training or 
use of computer-based instruction (CBI)/computer- 
based training (CBT) (Woit and Mason, 2000; Tunc and 
Armstead (2001); Kaczmarczyk 2001) and the usage is 
either in the corporate workplace or the elementary /high 
school skills-based context. Recently, with the new 
emphasis on technology, increased band-width and the 
dominance of the Internet, university professors are 
incorporating some of this technology into their courses. 
On-line testing is easy to track and document (Roberts 
2000; Tunc and Armstead 200 1 ; Woit and Mason 2000) 
and can be used to establish baselines for future 
assessment. Feedback mechanisms can be incorporated 
into the on-line testing (Gibson, Tesone, Blackwell 
2001) and different learning styles of the students can be 
accommodated. Therefore, this mode and process of 
testing or assessment may be more effective and 
efficient for both faculty and students. 

Dottie Natal (1998, 7-9) of Imagen Multimedia 
Corporation presented the advantages and disadvantages 
for both instructors and students of on-line testing at the 
Technology Education Conference in Santa Clara. 
Advantages or benefits for students are: 

• “Any time, any place” testing 

• Support for ESL (English is a second language) 
students 

• Support for students with disabilities 

• Valuable class time not used for exams 

• Depending upon the type of exam — immediate 
feedback may be available 

• Access to tools they are comfortable with 

Drawbacks for student include: 

• Procrastination - last minute test taking 

• Availability of computers 

• Lack of computer skills 

• Inability to use their standard test-taking skills 
(answering the hardest first, sure answers first....) 

Some benefits for the instructor are: 

• Decreased record-keeping time 



• Reusable test questions 

• Easier reading than hand-written 

• Increased teaching time 

• Better test design — for different learners 

• More frequent assessments 

• Better statistical feedback 

Drawbacks for instructors are: 

• Security problems 

• Unexpected results with regard to writing standards 

• Debugging problems with feedback or test 
administration 

• Hardware problems 

Other research has provided additional instructor 
enhancements to include immediate assessment, there- 
fore, enabling the instructor to review areas particularly 
challenging for individual students based upon on-line 
testing scores. This in turn provides class time for 
discussion of more difficult topics, instead of 
administering quizzes and exams (Woit and Mason 
2000; Tunc and Armstead 2001; Kaczmarczyk 2001). 

On-line testing has been identified with distance 
education courses. These are courses in which students 
enrolled do not attend class, do not accomplish the 
related tasks within a set time frame and can be at any 
location. Many of these students have computers at 
home, or have access at the workplace and have 
mastered the software. Why not use this mode of 
assessment for students enrolled in traditional face-to- 
face university courses? 

Universities provide access to computer labs throughout 
campus and in the dormitories. Some universities 
require computer ownership for admission. Universities 
are providing wireless connections to email and the 
Internet enabling laptops to be used anywhere on 
campus. In one scenario, on-line testing might be 
administered in a controlled setting, with-in one of the 
computer labs. 

One university experimented with installing a fully- 
outfitted computer lab for quiz taking. A proctor was 
hired to monitor testing activities. Students reserved a 
computer for completing the required quiz (Tunc and 
Armstead 2001). However, there is always the danger 
of the system being unable to accommodate users 
accessing the system at exactly the same time; each 
system has its own saturation point which must be taken 
into account. 
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Another scenario might have all students taking the test 
at the same time, but from any location they choose. 
This strategy not only raises overioading-the-system 
issue, but also monitoring the students while they are 
taking the test. Are they using books, documents, the 
Internet? Does the instructor care as long as the student 
is learning while they are completing the exam? Who is 
really taking the test? Will the technology create new 
methods of cheating? 

A third scenario is any time, any place, without 
restraints. This probably eliminates the system-overload 
issue, but not the other issues. Which delivery mode is 
optimal depends upon the preferences of the faculty and 
students and the university’s level of technology and 
support for that technology. 

LEARNING STYLES 

Much of the discussion surrounding alternative testing 
implementation centers around recent readings of 
various learning styles and their impact on testing and 
test results. Considerable research exists in tacit and 
explicit learning that will not be repeated here. 
Nevertheless, profound research conducted by Howard 
Gardner, author of Frames o/M/W(1983), which led to 
further research, and issuing of Multiple Intelligences 
(1993) identified educators’ oversight of various forms 
of intelligence that exists. 

Dr. Gardner’s work identifies more than one form of 
intelligence. Referred to as multiple intelligences, seven 
varieties of intelligence are identified by Dr. Gardner 
and his team of researchers. During the more than thirty 
years of work, psychologists working with all levels of 
education found that intelligence manifests itself in 
intrapersonal, interpersonal, spatial, linguistic, logical- 
mathematical, bodily-kinesthetic, and musical 
intelligence. It is stated that all individuals may possess 
more than one of the intelligences, and some, typically 
those who are mentally challenged, posses possibly just 
one of the identified intelligences. These multiple 
intelligences provide for a variety of perspectives on 
how information is learned. However, educators may 
fail to test more than one of these intelligences. 

Currently, assessment of learning is conducted towards 
the logical-mathematical intelligence as specified in 
Multiple Intelligences (1993). This assessment style is 
derives from Binet-Spearman formulation of evaluation. 
In the Binet-Spearman model, “the individual is tested 
in isolation” (Gardner 1983, 52). This approach is used 



when a student sits at a desk during a designated time 
period (that may not be the original class meeting time 
or regular class meeting room) and regurgitates data 
pertinent to the course. As Gardner states “the mind is 
a multifaceted, multi-component instrument, which 
cannot in any legitimate way be captured in a single 
paper and pencil-style instrument” (1993, 70) and 
further that “members of Mensa are expert in nothing — 
except in taking tests of intelligence” (1993, 53). These 
claims appear to be supported by very successful 
students who may not do well on exams; they were not 
the best test takes. 

The question that begs to be answered is ‘why do we 
continue to assess students in traditions set long ago?’ 
There is a greater understanding of learning styles which 
is not extended to the ability to adequately assess 
student grasp of information. 

ON-LINE TESTING IMPLEMENTED 

On-line testing was implemented in two different 
traditional fact-to-face courses. In one course, Business 
Data Communications, weekly quizzes forced students 
to read the text, become well prepared for class 
discussion and stay up-to-date with the course materials. 
In the other course, Management Information Systems 
Technology (MIST), an on-line test was used for the 
midterm with one section, while the other section took 
the exam in class. Students in both MIST sections were 
given the option for an on-line final exam or an in-class 
final exam. 

Both instructors used an on-line, Internet-based course 
management system. The system is used throughout the 
university in many different types of courses. All the 
students in the Data Communications course had used 
the software in at least one previous course. It was new 
for some students in the MIST course. The software 
provides automatic grading of certain types of questions 
and automatic posting of grades. Faculty can design the 
exam in any format. The easiest design may be 
true/false and multiple choice, but essay questions are 
accommodated. Immediate feedback on an individual 
answer is not a feature of the software; after the test is 
completed the student receives feedback on the entire 
exam, the software generates a comparison page of 
student answers and correct answers. 

The data communications course is an elective in the 
Information Systems field, the students are Juniors or 
Seniors, and all of the students have reasonable, if not 
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excellent computer skills and should all be familiar with 
the on-line course management program used for 
quizzes. In this course, valuable face-to-face in-class 
time was not spent on the weekly quizzes. The 
professor prepared only true/false and multiple choice 
questions based on the text material. The exam 
preparation time is equivalent to that for an on-line 
exam consisting of the same types of questions. The 
real time savings is in the auto-grading feature of the 
software. Exams were posted each week and the 
students had the entire week to complete the exam. The 
students had one hour to complete the exam; the 
software keeps track of the length of time spent on-line 
in the testing area. During class, following each exam, 
time was spent discussing/arguing about the exam 
questions, thereby reinforcing important concepts. 
Since extra points were earned if their arguments were 
successful, this strategy was a sure way to get students 
to participate in class discussions and stay current with 
the reading material. The students provided feedback on 
their experience. 

• The quizzes were a good learning tool and should be 
expanded 

• The quizzes forced me to stay current with the 
readings 

• The quizzes formed me to focus on the materials and 
class — usually I do not read the book before studying 
for mid-terms 

• I learned more because of the weekly quizzes 

One student commented that because the quizzes were 
administered on-line there was no reason to attend class. 

The MIST course is an introductory core Information 
Systems course and most of the students are 
Sophomores. Many of the students have used the on- 
line course management system before entering the 
course, some have not. However, all requirements and 
assignments for the course are available only on the on- 
line system and by the mid-term, each student should 
have experience with the system. The same instructor 
taught two sections of the MIST course; one section 
completed the mid-term during class time and the other 
section used the on-line system. The on-line students 
were on a limited time schedule, similar to those in the 
classroom. There is a built-in utility in the on-line 
system that releases the test materials at a certain time 
and then makes it unavailable after an elapsed period, 



however, the students were in the location of their 
choosing. Therefore, both sets of student had the same 
amount of time for completing the exam. The tests 
consisted of fifty multiple choice questions and nineteen 
short answer questions. The exam covered all materials 
up to the time of assessment. Both sets of students were 
advised to use textbooks, lecture notes, class papers and 
classmates in formulating their response. Surprisingly, 
exam results were slightly higher for the in-class test 
takers than forthe on-line test takers (Table 1). 



TABLE 1 

GRADE FREQUENCIES 





In-class: 
N = 29 


On-line: 
N = 31 


Mean 


97.7 


95.8 


Median: 


98.4 


96 


Mode: 


98.4 


98 


Std Dev. 


1.6 


2.36 


Min/Max 


93.6/100 


88/98 



There were 31 on-line test takers in one section of the 
undergraduate course and twenty-nine test takers in the 
in-class test-takers. Both sections were given the same 
test-taking directions. In-class test takers were asked to 
leave the room if they wanted to discuss the exam with 
classmates. Both exam periods were limited to the 
university requirements of 1 hour and fifty minutes. 

Both sections had the option to take the exam as either 
in-class, on-line, or a mix. Interestingly, students in one 
section chose to take the exam in-class, and the second 
section chose the on-line presentation. In neither section 
had students participated in an on-line testing 
experience. 

Data are limited to test scores. Demographic data on 
students was not collected in either testing environment. 
Nor was the amount of time taken for completion of the 
exam registered. While demographic and completion 
time data is available for the on-line students, these data 
were not collected for students completing the exam in- 
class. 

Comments from students taking the on-line test were 
overwhelmingly positive. Only two mishaps occurred. 
Both were satisfied by reinitiating the exam on-line for 
one student and providing a paper copy for the second 
student. The student opting to complete the exam on- 
line was able to answer all questions in the remaining 
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allotted time period. Since the questions were answered 
once already, the student had no trouble retaking the 
exam. The second student experienced no difficulties in 
completing the written exam. 

When final exam time arrived, both sections were again 
given the option to take the exam in-class, on-line, or a 
mix. The original in-class section opted for a mix of in- 
class and on-line with a majority taking the exam in- 
class. About one-third of students opted to complete the 
exam on-line. The original on-line class resoundingly 
opted for completing the final exam on-line. Part of this 
motivation may be a result of the exam scheduled on the 
very last exam date. This date may have interfered with 
summer plans. Taking the exam on-line allowed 
students to be at any location comfortable to the student. 

DISCUSSION 

On-line testing may add another level of complexity to 
the test-taking process. Either the instructor or the 
students may perceive on-line testing as another obstacle 
to conquer; in the students' mind it might be another 
deterrent to a good course grade. The level of technical 
skills of the instructor and their confidence in that skill 
are important factors in determining whether an attempt 
is made to incorporate any technology into the 
curriculum. The technical skill level and confidence of 
the student are also important factors. 

Does the on-line testing improve student learning and 
achievement? There was no statistically significant 
difference in the means for on-line exams and in-class 
exams. Additionally, the means for the on-line exam 
were lower than the means for the in-class exam. 
Therefore, in this study, there is no improvement in 
student learning and achievement when an on-line exam 
was administered. 

Can/should instructors tailor tests to student’s 
preferences for learning? Can/should instructors offer 
several versions of the test and let the students select 
their preference — before they see the questions? This 
may give the students more control over their outcomes. 
This issue requires further study and research. 

Another consideration of students in determining 
whether to take the test in-class or on-line may be the 
accessibility of the instructor to answer questions. In 
both instances, students were assured of instructor 
availability for questions arising during the exam. In the 
instance of the mixed test takers (some in-class, some 



on-line) the instructor carried a cell phone set to vibrate. 
In the instance of all on-line test takers, the instructor 
remained in the office during the entire testing period 
thereby providing access through email or telephone. 

How does this affect the instructor’s workload? 
“Faculty may wish to practice on the actual systems the 
students will be using” (Loeding and Wynn 1999, 181). 
A dry run enables the faculty to anticipate problems 
with the system, the software or the exam itself. Faculty 
administering on-line testing may have to be more 
flexible than those in a traditional classroom situation. 
Technical difficulties may negatively influence course 
evaluations at the end of the semester and can impact the 
perception of the entire course. In the two instances 
where the on-line exam failed, the instructor was 
flexible in allowing the students to retake the test in the 
on-line environment or to take the exam in a classroom 
environment. Although the task of grading may be 
eliminated for certain types of exams, this flexibility and 
additional access may increase the faculty’s workload. 

The concern about students copying or cheating is 
always present. Who is taking the exam and with whom 
are they taking it? The concept of cheating has long 
been a concern where there is no one monitoring the 
exam. Dr. Jerry Harvey (1988) claims that there is ‘no 
such thing as cheating.’ What Dr. Harvey referred to is 
the realization that learning occurs in various forms 
inside and outside the classroom. Dr. Harvey believes 
that collaboration is instrumental in the leaning process; 
it facilitates and enhances learning. He recognizes this 
need to accommodate various learning styles as an 
extension of future requirements in the chapter 
“Encouraging Future Managers to Cheat” in The Abilene 
Paradox (1988). In this chapter, Dr. Harvey acknow- 
ledges that once students graduate they will not work 
alone but with teams of people. He identifies areas of 
benefit in allowing students to work together in 
completing assessments as “providing a model of how 
work really gets done” (Harvey 1988, 124) and to do 
otherwise “thwarts the expression of synergy” (Harvey 
1988, 125). By providing those skills for which the 
student is best able to provide for the success of the 
team, the concept of conducting on-line testing may 
support positive interaction and building of relationships 
towards conclusion of a task. 

If Dr. Harvey’s claims are accepted, cheating is one less 
item of concern. Furthermore, as documented by 
Kaczmarczyk (2001) cheating as a phenomena during 
on-line exams has not been sufficiently documented. 
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Therefore, instances of cheating in distance or non- 
monitored test taking may not be an issue. In the MIST 
exam scenario, the instructor encouraged both classes to 
use class notes, lecture slides, and each other for 
formulating responses to questions. The reality is 
probably time constraints inhibited much if any 
conversation. 

Thunderstorms and power outages can adversely affect 
transmission of the test. Technical problems may make 
it impossible for a student to complete the exam in a 
timely fashion. If the exam is in a controlled computer 
laboratory setting, the system may not be able to handle 
20 to 30 attempts to access it at the same time. If the 
exam is truly an “anyplace” exam, bandwidth may 
influence the assessment outcomes. Students must be 
very knowledgeable of the computer system they are 
choosing. 

On-line testing should only be one component of the 
evaluation of the student (Gibson, Tesone, Blackwell 
2001). Other activities used to asses learning outcomes 
might be assignments, participation credit, projects, 
papers or case studies. 
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